Significance of relationships discovered in a corpus

ABSTRACT

Certain relationships representing material insights are identified from among a set of discovered relationships. Cognitive discovery of relationships in a knowledge base, or corpus, are ranked according to one or more metrics indicative of material insights, including recentness and degree of alignment.

BACKGROUND

The present invention relates generally to the field of cognitivecomputing, and more particularly to knowledge base development.

Cognitive computing is a field of artificial intelligence whichgenerally attempts to reproduce the behavior of the human brain.Cognitive computing refers to computing methods that are neither linearnor deterministic, but rather learn and interact with people in order toextend natural human function. Cognitive systems can perform a widevariety of tasks utilizing known artificial intelligence-based conceptssuch as natural language processing, information retrieval, knowledgerepresentation, automated reasoning, and machine learning.

Cognitive computing systems often discover numerous relationships fromanalysis of a body of knowledge, knowledge base, or corpus. Cognitivesystems are also good at highlighting the strongest relationships, suchas relationships to Ford Motor Company include: (i) Henry Ford, and (ii)Model T. (Note: the term(s) “Ford Motor Company” and/or “Model T” may besubject to trademark rights in various jurisdictions throughout theworld and are used here only in reference to the products or servicesproperly denominated by the marks to the extent that such trademarkrights may exist.)

Natural language processing (NLP) is a field of computer science,artificial intelligence, and linguistics concerned with the interactionsbetween computers and human (natural) languages. Generally speaking, NLPsystems use machine learning to analyze and derive meaning from textualcorpora (that is, sets of textual content). Many known NLP systemsannotate textual corpora (also referred to simply as “corpora,” or, inthe singular, a “corpus”) with information that may be helpful inunderstanding and/or interpreting the text. For a Definition of“annotate,” see the Definitions sub-section of the Detailed Descriptionsection.

SUMMARY

A method, computer program product, and computer system includes:identifying a focus within a natural language question; mining a firstbody of information for disclosure of entities within the first body ofinformation; determining relationships among the entities based on thefocus; generating a targeting document containing a first relationshipand a second relationship, the first and second relationships beingrelationships among the entities determined from the first body ofinformation; recording mining data for relationships in the targetingdocument in support of an importance criteria; ranking the firstrelationship with respect to the second relationship based on therecorded mining data and according to the importance criteria as a setof ranking data; and storing the targeting document including the set ofranking data and mining data in the first body of information foron-demand access during a question-answer session corresponding to thedomain of knowledge. The importance criteria is a first degree to whichfirst relationship is known and a second degree to which the secondrelationship is known according to a dictionary of commonly knownrelationships for the domain of knowledge. The first relationship beingranked as more important than the second relationship because the firstdegree is smaller than the second degree.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a schematic view of a first embodiment of a system accordingto the present invention;

FIG. 2 is a flowchart showing a first embodiment of a method performed,at least in part, by the first embodiment system;

FIG. 3 is a schematic view of a machine logic (for example, software)portion of the first embodiment system;

FIG. 4 is a screenshot view generated by the first embodiment system;and

FIG. 5 is a flowchart of a second embodiment of a method according tothe present invention;

FIG. 6 is a flowchart of a third embodiment of a method according to thepresent invention; and

FIG. 7 is a flowchart of a fourth embodiment of a method according tothe present invention.

DETAILED DESCRIPTION

Certain relationships representing material insights are identified fromamong a set of discovered relationships. Cognitive discovery ofrelationships in a knowledge base, or corpus, are ranked according toone or more metrics indicative of material insights, includingrecentness and degree of alignment. The present invention may be asystem, a method, and/or a computer program product. The computerprogram product may include a computer readable storage medium (ormedia) having computer readable program instructions thereon for causinga processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium, or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network, and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers, and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network, and forwards the computer readableprogram instructions for storage in a computer readable storage mediumwithin the respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computer,or entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture, including instructions which implement aspectsof the function/act specified in the flowchart and/or block diagramblock or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus, or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions, or acts, or carry out combinations of special purposehardware and computer instructions.

The present invention will now be described in detail with reference tothe Figures. FIG. 1 is a functional block diagram illustrating variousportions of networked computers system 100, in accordance with oneembodiment of the present invention, including: targeting sub-system102; client sub-systems 106, 108; domain sub-systems 104, 112; knowledgebases 105, 113; cognitive discovery sub-system 110; communicationnetwork 114; targeting computer 200; communication unit 202; processorset 204; input/output (I/O) interface set 206; memory device 208;persistent storage device 210; display device 212; external device set214; random access memory (RAM) devices 230; cache memory device 232;targeting program 300; and relationship store 305.

Sub-system 102 is, in many respects, representative of the variouscomputer sub-system(s) in the present invention. Accordingly, severalportions of sub-system 102 will now be discussed in the followingparagraphs.

Sub-system 102 may be a laptop computer, tablet computer, netbookcomputer, personal computer (PC), a desktop computer, a personal digitalassistant (PDA), a smart phone, or any programmable electronic devicecapable of communicating with the client sub-systems via network 114.Program 300 is a collection of machine readable instructions and/or datathat is used to create, manage, and control certain software functionsthat will be discussed in detail below.

Sub-system 102 is capable of communicating with other computersub-systems via network 114. Network 114 can be, for example, a localarea network (LAN), a wide area network (WAN) such as the Internet, or acombination of the two, and can include wired, wireless, or fiber opticconnections. In general, network 114 can be any combination ofconnections and protocols that will support communications betweenserver and client sub-systems.

Sub-system 102 is shown as a block diagram with many double arrows.These double arrows (no separate reference numerals) represent acommunications fabric, which provides communications between variouscomponents of sub-system 102. This communications fabric can beimplemented with any architecture designed for passing data and/orcontrol information between processors (such as microprocessors,communications and network processors, etc.), system memory, peripheraldevices, and any other hardware component within a system. For example,the communications fabric can be implemented, at least in part, with oneor more buses.

Memory 208 and persistent storage 210 are computer readable storagemedia. In general, memory 208 can include any suitable volatile ornon-volatile computer readable storage media. It is further noted that,now and/or in the near future: (i) external device(s) 214 may be able tosupply, some or all, memory for sub-system 102; and/or (ii) devicesexternal to sub-system 102 may be able to provide memory for sub-system102.

Program 300 is stored in persistent storage 210 for access and/orexecution by one or more of the respective computer processors 204,usually through one or more memories of memory 208. Persistent storage210: (i) is at least more persistent than a signal in transit; (ii)stores the program (including its soft logic and/or data), on a tangiblemedium (such as magnetic or optical domains); and (iii) is substantiallyless persistent than permanent storage. Alternatively, data storage maybe more persistent and/or permanent than the type of storage provided bypersistent storage 210.

Program 300 may include both machine readable and performableinstructions, and/or substantive data (that is, the type of data storedin a database). In this particular embodiment, persistent storage 210includes a magnetic hard disk drive. To name some possible variations,persistent storage 210 may include a solid state hard drive, asemiconductor storage device, read-only memory (ROM), erasableprogrammable read-only memory (EPROM), flash memory, or any othercomputer readable storage media that is capable of storing programinstructions or digital information.

The media used by persistent storage 210 may also be removable. Forexample, a removable hard drive may be used for persistent storage 210.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer readable storage medium that is also part of persistent storage210.

Communications unit 202, in these examples, provides for communicationswith other data processing systems or devices external to sub-system102. In these examples, communications unit 202 includes one or morenetwork interface cards. Communications unit 202 may providecommunications through the use of either, or both, physical and wirelesscommunications links. Any software modules discussed herein may bedownloaded to a persistent storage device (such as persistent storagedevice 210) through a communications unit (such as communications unit202).

I/O interface set 206 allows for input and output of data with otherdevices that may be connected locally in data communication withcomputer 200. For example, I/O interface set 206 provides a connectionto external device set 214. External device set 214 will typicallyinclude devices such as a keyboard, keypad, a touch screen, and/or someother suitable input device. External device set 214 can also includeportable computer readable storage media such as, for example, thumbdrives, portable optical or magnetic disks, and memory cards. Softwareand data used to practice embodiments of the present invention, forexample, program 300, can be stored on such portable computer readablestorage media. In these embodiments the relevant software may (or maynot) be loaded, in whole or in part, onto persistent storage device 210via I/O interface set 206. I/O interface set 206 also connects in datacommunication with display device 212.

Display device 212 provides a mechanism to display data to a user andmay be, for example, a computer monitor or a smart phone display screen.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of the presentinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus the presentinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

Targeting program 300 operates to identify important relationshipdiscoveries according to a pre-defined metric for comparing discoveredrelationships in a knowledge base. The comparison of relationshipssupports a ranking process, the highly ranked, or top-ranked,relationships being the ones that are most likely to bring new insightsto the field of study represented by the knowledge base.

Some embodiments of the present invention recognize the following facts,potential problems and/or potential areas for improvement with respectto the current state of the art: (i) while relationships identified, or“discovered,” by cognitive systems are often accurate, the relationshipsmay reflect common knowledge rather than a material insight into afield; (ii) current cognitive computing systems are able to uncoverhidden relationships, but depend on human analysts to determine thesignificance of the discovered relationships; and/or (iii) cognitivecomputing systems often uncover numerous relationships in a givenknowledge base.

Some embodiments of the present invention are directed to a method foridentifying material insights in the set of automatically discoveredrelationships by a cognitive system. Some embodiments of the presentinvention are directed to calculating a “degree of alignment” based onthe amount of evidence there is to identify the relationship.Alternatively, the number of evidentiary documents that are the basisfor identifying the relationship are considered for calculating thedegree of alignment.

Considering the degree of alignment, the cognitive system can then be“controlled” to show discovered relationships corresponding to eithermore or less evidence depending on user preference. Where less evidenceis the basis of a discovered relationship, less obvious materialinsights are identified from among the more obvious insights that may bediscovered (with more certainty).

Some embodiments of the present invention track the degree of alignmentfor discovered relationships over time according to the latest state ofthe cognitive system as the knowledge base develops.

Some embodiments of the present invention are directed to an algorithmto determine the materiality of relationships identified by cognitivediscovery solutions. The materiality being, in some embodiments,described by the “degree of alignment” with the evidence. Havingdetermined the materiality, the output of the cognitive system isprocessed and prioritized according to a ranking of materiality.Further, in some embodiments, the results are presented to a user in aUI (user interface) showing the decisions and/or evidence that supportsthe materiality ranking.

FIG. 2 shows flowchart 250 depicting a first method according to thepresent invention. FIG. 3 shows program 300 for performing at least someof the method steps of flowchart 250. This method and associatedsoftware will now be discussed, over the course of the followingparagraphs, with extensive reference to FIG. 2 (for the method stepblocks) and FIG. 3 (for the software blocks). FIG. 4 depicts an examplescreenshot displayed during operation of an embodiment of the presentinvention.

Processing begins at step S255, where ingest module (“mod”) 355 ingestsa knowledge base, or corpus, or a particular domain of knowledge ortechnical field. In this example, the ingest module ingests knowledgebase 105 (FIG. 1), which includes data regarding the domain of toys.Alternatively, the ingest module ingests multiple knowledge bases, suchas both knowledge bases 105 and 113.

Processing proceeds to step S260, where question mod 360 receives aquestion. In this example, the question mod receives a question from auser via a user interface (UI) displayed on display device 212 (FIG. 1).The user input “who makes the top selling toys” as the question.Alternatively, the question is generated as a set of training questionsand input for receipt by the question mod according to a manner nowknown, or to be known in the future.

Processing proceeds to step S265, where parse mod 365 parses thequestion received at step S260 into a natural language processing (NPL)structure including a focus. The focus is, in some embodiments, theprimary search query developed from the question received. Based on theabove stated question, the focus is determined to be “Toys.”Additionally, the NPL structure oftentimes includes a lexical answertype (LAT). An LAT is a word or noun phrase in the received questionthat indicates the type of answer without requiring any semanticanalysis. If the LAT were applied in this question, the type of answeris “who makes X.” While cognitive computing systems may provideadditional NPL structure, such as the LAT, having a capability ofparsing a natural language question to identify a focus of the questionis sufficient for practicing various embodiments of the presentinvention.

Processing proceeds to step S270, where focus mod 370 identifiesentities, or objects, in the knowledge base and their correspondingrelationships based on the focus derived from parsing the question. Inthis example, the relationship between manufactured toy products and themanufacturer are identified based on the focus of the question receivedin step S260.

Processing proceeds to step S275, where ranking mod 375 ranks theidentified relationships according to an importance criteria. Becausethe question did not recite a year in which the top selling toys were tobe identified, the importance criteria is “recentness.” Accordingly, theranking mod operates to determine a most recent indication of a topselling toy and corresponding manufacturer. As discussed in more detailbelow, one or more of various tests are applied to the identifiedrelationships including: (i) temporal test; (ii) evidence in domaintest; (iii) evidence outside of domain test; (iv) age test; (v) type ofevidence test; (vi) user criteria test; and (vii) known relationshiptest.

Processing ends at step S280, where report mod 380 generates a report ofthe ranked relationships according to the importance criteria. In thisexample, a report is displayed via the UI to the user for processing.For example, the user may select a top 5 relationships for printing tohard copy. Alternatively, the report is distributed to a pre-defined setof email addresses, or other contact information. Alternatively, theuser is provided with input options to refine the original question.

While processing ends in this discussion at step S280, processing mayreturn to other steps, such as step S265, when the user is provided anoption to restate the question in light of the reported data.

Further embodiments of the present invention are discussed in theparagraphs that follow and later with reference to FIGS. 5 and 6.

Some embodiments of the present invention apply an algorithm that usesthe output of existing cognitive discovery solutions, such as WatsonDiscovery Advisor, to identify a superset of relationships. (Note: theterm(s) “Watson” and/or “Discovery Advisor” may be subject to trademarkrights in various jurisdictions throughout the world and are used hereonly in reference to the products or services properly denominated bythe marks to the extent that such trademark rights may exist.) Usingthis output, the relationships are ranked in order of importance basedon a variety of criteria discussed in more detail below. The rankingcalculation is an average of the weight given to criterion in the set ofcriteria by the user and multiplied by a ranking factor that isnormalized for each step of the algorithm. In some embodiments, forthose results that fall below a threshold rank, the user has the optionof eliminating them from the results set.

Some embodiments of the present invention rely on a cognitive discoverysolution to identify relationships based on a query by the user. In thisexample, the relationships are output to a format, such as XML, whichcan be understood by the targeting program, such as targeting program300 (FIG. 1). Further, in this example, the relationship information isparsed into an operational store. Once in the operational store, thealgorithm is, in this example, triggered by a user request.Alternatively, the algorithm is triggered by a pre-defined schedule.Alternatively, the algorithm is triggered automatically when a new queryis run.

Some embodiments of the present invention are directed to a system thatdraws from a historical view of discovered relationships to highlightonly those relationships within a set of discovered relationships thatare, for example, (i) new, (ii) recently changed, and/or (iii) recentlybeing mentioned significantly more than others (suggesting newinformation). In that way, the output of the cognitive system isprioritized according to content recentness, or timeliness, with respectto the corpus of knowledge being studied.

Some embodiments of the present invention are directed to the study ofthe age of discovered relationships based on previous mentions. If adiscovered relationship is brand new, it is flagged as such and itsrelevance ranking is increased accordingly. If a discovered relationshippre-exists, but has changed recently, the original relationship and therecent changes are recorded. In some embodiments, all discoveredrelationships within a given corpus are checked for “frequency ofmentions” according to a timeline. The mention frequency data is matchedagainst past mention frequency results to observe trends. If amention-frequency trend is increasing, then the relationship isconsidered to be timely as to relevance and it is flagged for review,for example, the relationship is recorded as timely. Alternatively, therelationship is tagged with meta-data for later recovery and/or search.

Some embodiments of the present invention operate in the following way.When the algorithm is triggered, the user is prompted to provideweighting information for each algorithm sub-component (for example, arange from 0-6). The algorithm processes each step as an independentthread and performs the ranking calculation when the steps are complete.The results are then returned to the user in ranked order with evidenceattached for the ranking.

An algorithm according to some embodiments of the present invention isdescribed in the next few paragraphs. For each discovered relationshipthe follow tests are applied: (i) a test of the newness, or recentness,of the relationship (based on a semantic search of available bodies ofknowledge, the relationship is determined to be new for the domain) (forexample, Ford ->Henry Ford ->Model T may be a new relationship in apolitical science domain, but not for an automotive use case) (brand newrelationships are ranked higher than older relationships); (ii) thenumber of mentions of the discovered relationship in the specific bodyof knowledge studied; (iii) the number of mentions of the discoveredrelationship in general body of knowledge such as Wikipedia; (iv) thetype and/or provenance of evidence supporting the discoveredrelationship; (v) the age of the relationship (in general, a higherranking will be granted to newer relationships according to a slidingtime scale appropriate to the subject matter; (vi) appearance of therelationship in a dictionary of known or common knowledge; (vii)user-defined criteria, such as a concept, keyword, or phrase whosepresence has the effect of increasing/decreasing the rank or removingthe relationship from review; and/or (viii) output format.

Another algorithm, which focuses on the recentness of discoveredrelationships includes: (i) a test of the relationship for changes fromthe original, particularly recent changes from earlier relatedstatus(es) (for example, GM ->Henry Ford ->Model T is a discoveredrelationship that is changed with respect to the above-citedrelationship); and/or (ii) a count of relationship mentions isdetermined from the studied knowledge corpus, the count is stored as aninteger and compared to past mention counts to identify trends inrelationship mentions.

Some embodiments of the present invention test newly discoveredrelationships to understand if the discovered relationship is accountedfor as a “changed relationship” with respect to an “original,” earlierrecitation, of a relationship, or an entirely new relationship. Thedetermination is, in some embodiments, performed by a secondary routine,which triggers a cognitive discovery of the relationship of new andoriginal concept. If there is a relationship showing a transition overtime between a “new” and a corresponding original concept, then there isan assumption that a material change has occurred. This “new” conceptmay appear to be a semantic variant, which is also material. Forexample, Ford Motor Company -->Henry Ford -->Model T is a semanticvariant to Ford ->Henry Ford ->Model T. Both situations may be flaggedin the UI as being material.

In some embodiments of the present invention, mention trend changes,particularly increases, are recorded as material. For example, Ford->Henry Ford ->Model T registered 140 mention as of yesterday, but todayFord ->Henry Ford ->Model T registered 10,000 mentions. This change isflagged as likely being a material relationship according to recentnessconsiderations. Additionally, threshold values may be established by auser for flagging a relationship as material based on a pre-determinedincrease in mentions.

Some embodiments of the present invention are directed to a specialevaluation of the knowledge corpus being studied to understand a basisfor changes in the mention-frequency counts. If an increase or decreasein mention counts is due to the inclusion, or exclusion, of a particulardocument this is also considered to be material.

Some embodiments of the present invention output components of the testaccording to a rank from 0-1. User-defined weights may, in some example,be required to combine to be 100%. Newness, or recentness, ranks high onthe list based on the assumption that new relationships in the domainare always material. Count (integer) of mentions from output (the higherthe count the higher the rank). Count (integer) of mentions from outputagainst a general body of knowledge, where the higher count results in ahigher rank. Evidence sources are matched against a pre-determinedranking of veracity of the sources from an impartial third party ratingagency. The more authoritative evidence receives a higher rank. Tracksagainst a dictionary of original timestamp of when the relationship wasfirst found. Older relationships rank lower. Match against a dictionaryof commonly known relationships for a domain. A matched relationshipindicates common knowledge and is ranked lower or completely removedfrom review. The UI accepts concepts, keywords, and/or phrases definedby the user for the algorithm tests against a specific and/or generalbody of knowledge.

Additional ranking considerations, focused on recentness, support outputincluding: (i) new relationship(s) plus supporting evidence; (ii)original relationship(s), corresponding changed relationship(s), orsemantic variant(s) of the original relationship(s), and supportingevidence; and/or (iii) a count for the number of mentions organized by“past” and “present” mentions, and supporting evidence (corpus changesmay also be output as-appropriate).

For each result from the relationship discovery, a calculation (1) fromthe segments 0-6 is performed:

Sum(User Defined Weight*Rank Integer)   (1)

Results are returned on a scale from 0-100 with higher being better. Anormalization step is needed to properly score up to 100, which is aperfect score. Users can configure a threshold materiality for removingcertain relationships via, for example, a UI.

Some embodiments of the present invention are directed to a method toassign a value to discovered relationships in an information handlingsystem capable of answering questions comprising: ingesting, by asystem, a body of information, or corpus, for a domain; receiving, bythe system, a question; applying natural language processing (NLP), bythe system, to parse the question into an NLP structure comprising alexical answer type (LAT) and a focus, or target; mining the body ofinformation to identify entities and relationships between the entitiesbased on the focus; and ranking, that is, assigning a value to,relationships according to an importance criteria, or value, comprisingweighting for age in the domain, weighting age in an general, usuallypublic, body of knowledge, and weighting for references.

Some embodiments of the present invention are further directed toproviding a user interface (UI) that allows a user to adjust theimportance criteria with additional detailed weightings comprisingconcepts, keywords, and phrases; and responsive to the user utilizingthe UI, applying the detailed weighting to relationships to adjust theranking, and displaying the relationships ordered by ranking.

Some embodiments of the present invention are directed to a UI thatfurther comprises: displaying the relationships ordered according toaspects weighted by, for example, the age of the relationship, corpuscounts, external counts, evidence source reliability, and user-definedcriteria.

FIG. 5 shows flowchart 500 depicting a second method according to anembodiment of the present invention. This method, which establishes arun-time results relevancy ranking, will now be discussed, over thecourse of the following paragraphs.

Processing begins at step S502, where a cognitive program is executed toidentify, or discover, relationships based on a query submitted by auser. In this example, the identified relationships are output as an XMLdocument.

Processing proceeds to step S504, where the XML document is converted toa targeting document in a preferred format for processing.

Processing proceeds to step S506, where the targeting document havingthe identified relationships is stored for relationship analysis.

Processing proceeds to step S508 where a relationship analysis programis initialized for analyzing the targeting document generated in stepS504.

Processing proceeds to step S510, where the process of weighting thevarious aspects of the targeting document with respect to the discoveredrelationships is implemented. In this example, sub-steps S511-S517comprise the weighting process.

For each sub-step test, the results are recorded for each discoveredrelationship. The combined results of the sub-step tests provide anindicator as to the significance of each discovered relationship.

The tests in this example are as follows: temporal test S511; evidencein domain test S512; evidence outside of domain test S513; age testS514; type of evidence test S515; user criteria test S516; and knownrelationship test S517. In the alternative, only one or fewer than allof the tests are performed to produce “combined” test results, or“weighted results.” That is, the test operate to establish a weightingamong the various relationships. The weighting being an indication as tothe significance, or materiality, of the insight involved in discoveringa given relationship.

Temporal test S511 operates to determine the time in which a discoveredrelationship is first identifiable according to the domain-specificcorpus.

Evidence in domain test S512 operates to determine a degree, or relativeamount, of evidence available in the domain-specific corpus to support adiscovered relationship.

Evidence outside of domain test S513 invokes secondary knowledge basesoutside of the focus domain. When such non-domain corpus is available,this test operates to determine a degree, or relative amount, ofevidence available in that secondary knowledge base.

Age test S514 operates to determine the duration of time that thediscovered relationship was known according to the domain-specificcorpus.

Type of evidence test S515 operates to sort the various evidence datapoints according to a type of evidence. For example, some evidence isgenerated by a third-party and other evidence is generated by the partyof interest, or subject of the data. That is, the source of the evidenceis one way to classify a “type” of evidence.

User-criteria test S516 is a user-generated criteria specific to thequestion at hand. The user-criteria test operates to further refine theranking discovered relationship analysis according to a custom criteria.

Known relationship test S517 operates to assign a lower ranking torelationships that are determined to be known in the focus domain. Inthat way, the material insights are higher ranked than those potentialmaterial insights that are already known within the focus domain.

Processing proceeds to step S520, where the rank is calculated toestablish an order of significance among a set of discoveredrelationship included in the targeting document.

Processing ends at step S522, where the rankings are stored for laterreference.

FIG. 6 shows flowchart 600 for determining relevant relationships basedon temporal continuance according to an embodiment of the presentinvention. This method will now be discussed, over the course of thefollowing paragraphs.

Processing begins at step S602, where a document is processed. In thisstep, a designated document is processed for relationship data to beincluded in a corpus of data for a question-answering system. The datais processed in that it is translated and/or formatted as-needed forinterpretation and/or analysis.

Processing proceeds to step S604, where relationships among entities aspresented within the document are identified and extracted. Thisactivity may also be referred to a discovering relationships within thedocument.

Processing proceeds to step S606, where meta-data is extracted from thedocument. The meta-data in the document is associated with thediscovered relationships. The meta-data may provide helpfulclassification of the discovered relationships, including: (i) date ofpublication; (ii) date of draft; (iii) originator (or owner) of thedocument; (iv) rating of the author; (v) source of document (government,news service, technical journal); and/or (iv) revision history.Depending on the source and/or other meta-data the accuracy and/orreliability of the information may be weighted. The more often arelationship is detected in reliable, or trusted, source documents, themore accurate and/or reliable the relationship becomes. In someembodiments one source is deemed more-trusted than or sources. Amore-trusted source has a higher weight than the weight of an averagesource, while a less-trusted source has a lower weight than the weightof an average source.

Processing proceeds to decision step S608, where a determination is madefor each discovered relationship as to whether the relationship is newto the database, or store, where earlier discovered relationshipsassociated with the particular domain are recorded, or registered (inthis example). If a discovered relationship is new, then processingfollows the “Yes” branch to step S610. If a discovered relationship isnot new, processing follows the “No” branch to step S614. In thisexample, where no new relationship is found among any of the discoveredrelationships, processing follows the “No” branch. For a set of newrelationships, processing follows the “Yes” branch, where therelationship data is registered. This example describes batch processingof the discovered relationships. Alternatively, for each discoveredrelationship, it is determined whether or not it is a new relationship.Processing proceeds through the following steps, returning to S608 untilall discovered relationships are addressed.

For a new relationship, processing proceeds to step S610, where the newrelationship is registered. In this example, registration includesstoring the relationship in a relationship store, such as relationshipstore 305 (FIG. 1) in the form of a triple. Alternative storage schemesmay apply, so long as the relationship is recorded for later reference.

Processing proceeds to step S612, where the meta-data extracted in stepS606 is registered. In this example, relationships and theircorresponding meta-data are associated together in a relationship storefor later reference. Alternatively, a separate meta-data store recordsmeta-data registration. As stated earlier, the associated meta-dataprovides insight into temporal continuance of the discoveredrelationship.

Processing proceeds, whether from step S608 or from step S612, to stepS614, where decay of the discovered relationship(s) is computed. Inorder to establish temporal continuance, recentness, and/or degree ofalignment, the relative decay of discovered relationships providestimeliness data for ranking a set of relationships according to one ormore of those metrics.

Processing proceeds to step S616, where an importance level is computedfor the discovered relationship(s). An importance level for a givenrelationship depends largely on the metric used for comparingrelationships. Importance factors include temporal continuance,recentness, and/or degree of alignment. Some embodiments of the presentinvention recognize that a discovered relationship that is recent may bemore “important” than one that has been known for a long time. Further,a discovered relationship that is strongly aligned with generalknowledge in a particular domain is likely less important than one thatis less aligned with such knowledge.

Processing ends at step S618, where the results of decay and importancecomputations are stored. In this example, the discovered relationship,corresponding meta-data, and calculated markers such as decay andimportance are each stored in a relationship store for on-demand accessand analysis, for example, during a question-answer session by acognitive computing system.

FIG. 7 shows flowchart 700 for determining answers to a question rankedaccording to temporal continuance. This method will now be discussed,over the course of the following paragraphs.

Processing begins at step S702, where a natural language question isprocessed. In this example, a user inputs a natural language questioninto a question-answering system. The question is processed according toconventional methods where the question is parsed for characteristicelements such as lexical answer type (LAT) and question focus.

Processing proceeds to step S704, where the question-answering systemdetermines a set of appropriate answers to the question processed instep S702.

Processing proceeds to step S706, where the set of answers determined instep S704 are filtered according to the computed decay and/or importanceof corresponding relationships supporting one or more of the variousanswers.

By filtering the set of answers, the answer(s) provided are more likelyto provide insightful and more helpful information. In some embodimentsof the present invention, the filtering process yields a single highlyrelevant and insightful answer to the question processed in step S702.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) appliesto many types of cognitive discovery process, including those used infinancial services, healthcare, and culinary arts; (ii) makes adetermination as to the significance of discovered relationships; (iii)assists human analysts to process discovered relationships to increasethe speed of identification of material relationships worth even closeranalysis; (iv) precise targeting of relevant results; (v) applies atargeting algorithm that allows for a substantial reduction in researchtime; (vi) reduces the noise of the output; (vii) reduces potentialdistraction and/or fatigue of the human analyst; (viii) increased userconfidence in provided results; (ix) makes a determination as to thematerialness of relationships from the user's perspective; (x) makes adetermination as to the materialness of relationships or from apre-determined criteria; (xi) makes a determination as to thematerialness decay of relationships from a user's perspective; and/or(xii) makes a determination as to the materialness decay ofrelationships from per-determined criteria.

Some helpful definitions follow:

Present invention: should not be taken as an absolute indication thatthe subject matter described by the term “present invention” is coveredby either the claims as they are filed, or by the claims that mayeventually issue after patent prosecution; while the term “presentinvention” is used to help the reader to get a general feel for whichdisclosures herein that are believed as maybe being new, thisunderstanding, as indicated by use of the term “present invention,” istentative and provisional and subject to change over the course ofpatent prosecution as relevant information is developed and as theclaims are potentially amended.

Embodiment: see definition of “present invention” above—similar cautionsapply to the term “embodiment.”

and/or: inclusive or; for example, A, B “and/or” C means that at leastone of A or B or C is true and applicable.

User/subscriber: includes, but is not necessarily limited to, thefollowing: (i) a single individual human; (ii) an artificialintelligence entity with sufficient intelligence to act as a user orsubscriber; and/or (iii) a group of related users or subscribers.

Module/Sub-Module: any set of hardware, firmware and/or software thatoperatively works to do some kind of function, without regard to whetherthe module is: (i) in a single local proximity; (ii) distributed over awide area; (iii) in a single proximity within a larger piece of softwarecode; (iv) located within a single piece of software code; (v) locatedin a single storage device, memory or medium; (vi) mechanicallyconnected; (vii) electrically connected; and/or (viii) connected in datacommunication.

Computer: any device with significant data processing and/or machinereadable instruction reading capabilities including, but not limited to:desktop computers, mainframe computers, laptop computers,field-programmable gate array (FPGA) based devices, smart phones,personal digital assistants (PDAs), body-mounted or inserted computers,embedded device style computers, application-specific integrated circuit(ASIC) based devices.

Natural Language: any language used by human beings to communicate witheach other.

Natural Language Processing: any derivation of meaning from naturallanguage performed by a computer.

Annotate: to add information to text, including, but not limited to: (i)adding additional textual content, such as by linking, tagging (andpart-of-speech tagging), adding metadata, providing markup, indicatinglemma (or base forms of a word), labeling, commenting, or explaining;and/or (ii) modifying the text, such as by highlighting, underlining,bolding, italicizing, or parsing.

What is claimed is:
 1. A method comprising: identifying a focus within anatural language question; mining a first body of information fordisclosure of entities within the first body of information; determiningrelationships among the entities based on the focus; generating atargeting document containing a first relationship and a secondrelationship, the first and second relationships being relationshipsamong the entities determined from the first body of information;recording mining data for relationships in the targeting document insupport of an importance criteria; ranking the first relationship withrespect to the second relationship based on the recorded mining data andaccording to the importance criteria as a set of ranking data; andstoring the targeting document including the set of ranking data andmining data in the first body of information for on-demand access duringa question-answer session corresponding to the domain of knowledge;wherein: the importance criteria is a first degree to which firstrelationship is known and a second degree to which the secondrelationship is known according to a dictionary of commonly knownrelationships for the domain of knowledge; and the first relationshipbeing ranked as more important than the second relationship because thefirst degree is smaller than the second degree.
 2. The method of claim1, further comprising: parsing the natural language question to identifythe focus.
 3. The method of claim 1, wherein the importance criteria isfurther based upon an age of the first relationship compared to an ageof the second relationship.
 4. The method of claim 1, wherein theimportance criteria relates to a comparison between a first set ofevidence that identifies the first relationship with a second set ofevidence that identifies the second relationship.
 5. The method of claim4, wherein the comparison between the first set of evidence and thesecond set of evidence is based at least in part upon a number ofevidentiary documents within each of the first set of evidence and thesecond set of evidence.
 6. The method of claim 1, further comprising:registering the corresponding set of relationships in a relationshipstore having a set of relationships identified from the first body ofinformation.
 7. The method of claim 1 further comprising: reporting to auser a set of top-ranked relationships in the targeting document.
 8. Themethod of claim 1, further comprising: classifying an identifiedrelationship as a less-trusted fact; and modifying the classification ofthe identified relationship from the less-trusted fact to a more-trustedfact based at least in part upon a count of trusted sources that supportthe identified relationship.
 9. The method of claim 1, wherein thenatural language question is presented as a training questioncorresponding to a new document to be added to the first body ofinformation.
 10. The method of claim 9, further comprising:incorporating the new document into the first body of information toform a second body of information.
 11. The method of claim 1, furthercomprising: reporting the set of ranking data as input to determine arestated question.
 12. A computer program product comprising acomputer-readable storage medium having a set of instructions storedtherein which, when executed by a processor, causes the processor todetermine a significance ranking for discovered relationships by:identifying a focus within a natural language question; mining a firstbody of information for disclosure of entities within the first body ofinformation; determining relationships among the entities based on thefocus; generating a targeting document containing a first relationshipand a second relationship, the first and second relationships beingrelationships among the entities determined from the first body ofinformation; recording mining data for relationships in the targetingdocument in support of an importance criteria; ranking the firstrelationship with respect to the second relationship based on therecorded mining data and according to the importance criteria as a setof ranking data; and storing the targeting document including the set ofranking data and mining data in the first body of information foron-demand access during a question-answer session corresponding to thedomain of knowledge; wherein: the importance criteria is a first degreeto which first relationship is known and a second degree to which thesecond relationship is known according to a dictionary of commonly knownrelationships for the domain of knowledge; and the first relationshipbeing ranked as more important than the second relationship because thefirst degree is smaller than the second degree.
 13. The computer programproduct of claim 12, further comprising: parsing the natural languagequestion to identify the focus.
 14. The computer program product ofclaim 12, wherein the importance criteria is further based upon an ageof the first relationship compared to an age of the second relationship.15. The computer program product of claim 12, wherein the importancecriteria relates to a comparison between a first set of evidence thatidentifies the first relationship with a second set of evidence thatidentifies the second relationship.
 16. The computer program product ofclaim 15, wherein the comparison between the first set of evidence andthe second set of evidence is based at least in part upon a number ofevidentiary documents within each of the first set of evidence and thesecond set of evidence.
 17. A computer system comprising: a processorset; and a computer readable storage medium; wherein: the processor setis structured, located, connected, and/or programmed to run programinstructions stored on the computer readable storage medium; and theprogram instructions which, when executed by the processor set, causethe processor set to determine a significance ranking for discoveredrelationships by: identifying a focus within a natural languagequestion; mining a first body of information for disclosure of entitieswithin the first body of information; determining relationships amongthe entities based on the focus; generating a targeting documentcontaining a first relationship and a second relationship, the first andsecond relationships being relationships among the entities determinedfrom the first body of information; recording mining data forrelationships in the targeting document in support of an importancecriteria; ranking the first relationship with respect to the secondrelationship based on the recorded mining data and according to theimportance criteria as a set of ranking data; and storing the targetingdocument including the set of ranking data and mining data in the firstbody of information for on-demand access during a question-answersession corresponding to the domain of knowledge; wherein: theimportance criteria is a first degree to which first relationship isknown and a second degree to which the second relationship is knownaccording to a dictionary of commonly known relationships for the domainof knowledge; and the first relationship being ranked as more importantthan the second relationship because the first degree is smaller thanthe second degree.
 18. The computer system of claim 17, furthercomprising: parsing the natural language question to identify the focus.19. The computer system of claim 17, wherein the importance criteria isfurther based upon an age of the first relationship compared to an ageof the second relationship.
 20. The computer system of claim 17, whereinthe importance criteria relates to a comparison between a first set ofevidence that identifies the first relationship with a second set ofevidence that identifies the second relationship.